HORARY 

TECHNICAL  REPORT  SECTION 
naval  postgraduate  school 
MONTEREY.  CALIFORNIA  93940 


AN  APPLICATION  OF 
MULTI-ATTRIBUTE  UTILITY  THEORY: 
DESIGN-TO-COST  EVALUATION  OF  THE 
U.S.  NAVY'S  ELECTRONIC  WARFARE  SYSTEM 

DECISIONS  AND  DESIGNS  INCORPORATED 


Michael  L.  Hays 
Michael  F.  O'Connor 
Cameron  R.  Peterson 


ADVANCED  <m 
DECISION  TECHNOEOGY 

PROGRAM 


CYBERNETICS  TECHNOLOGY  OFFICE 

DEFENSE  ADVANCED  RESEARCH  PROJECTS  AGENCY 

Office  of  Naval  Research  • Engineering  Psychology  Programs 


The  objective  of  the  Advanced  Decision 
Technology  Program  is  to  develop  and  transfer 
to  users  in  the  Department  of  Defense  advanced 
management  technologies  for  decision  making. 

These  technologies  are  based  upon  research 
in  the  areas  of  decision  analysis,  the  behavioral 
sciences  and  interactive  computer  graphics. 
The  program  is  sponsored  by  the  Cybernetics 
Technology  Office  of  the  Defense 
Advanced  Research  Projects  Agency  and 
technical  progress  is  monitored  by  the  Office 
of  Naval  Research  — Engineering  Psychology 
Programs.  Participants  in  the  program  are: 

Decisions  and  Designs,  Incorporated 
The  Oregon  Research  Institute 
Perceptronics,  Incorporated 
Stanford  Research  Institute 
Stanford  University 
The  University  of  Southern  California 

Inquiries  and  comments  with 
regard  to  this  report  should  be 
addressed  to: 

Dr.  Martin  A.  Tolcott 
Director,  Engineering  Psychology  Programs 
Office  of  Naval  Research 
800  North  Quincy  Street 
Arlington,  Virginia  22217 

or 

Dr.  Robert  A.  Young 

Cybernetics  Technology  Office 
Defense  Advanced  Research  Projects  Agency 
1400  Wilson  Boulevard 
Arlington,  Virginia  22209 


The  views  and  conclusions  contained  in  this  document  are  those  of  the  author(s)  and  should  not  be  interpreted  as  necessarily 
representing  the  official  policies,  either  expressed  or  implied,  of  the  Defense  Advanced  Research  Projects  Agency  or  the  U.S. 
Government.  This  document  has  been  approved  for  public  release  with  unlimited  distribution. 


Unclassified 


SECURITY  CLASSIFICATION  OF  This  PACE  (Whan  Data  Entatad) 


REPORT  DOCUMENTATION  PAGE 

READ  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 

1 REPORT  NUMBER 

2.  GOVT  ACCESSION  NO. 

3.  RECIPIENT’S  CATALOG  NUMBER 

4.  TITLE  (and  Subtltta) 

An  Application  of  Multi-Attribute  Utility 
Theory:  Design- to-Cost  Evaluation  of  the  U.S. 

5.  TYPE  OF  REPORT  A PERIOD  COVEREO 

Technical  Report 
July  19 73- June  1974 

Navy's  Electronic  Warfare  System 

6.  PERFORMING  ORG.  REPORT  NUMBER 

DT/TR  75-3 

7.  AUTHORS 

Michael  L.  Hays,  Michael  F.  O’Connor,  and 
Cameron  R.  Peterson 

8.  CONTRACT  OR  GRANT  NUMBERf*; 

N00014-75-C-0426 

N00039-75-C-0084 

N00024-73-C-1060 

9.  PERFORMING  ORGANIZATION  NAME  ANO  ADDRESS 

Decisions  and  Designs,  Incorporated 
Suite  600,  8400  Westpark  Drive 
McLean.  Virginia  22101 

10.  PROGRAM  ELEMENT.  PROJECT,  TASK 
AREA  a WORK  UNIT  NUMBERS 

II.  CONTROLLING  OFFICE  NAME  ANO  ADDRESS 

Office  of  Naval  Research  (Code  455) 

12.  REPORT  DATE 

October,  1975 

800  North  Quincy  Street 
Arlington,  Virginia  22217 

11.  NUMBER  OF  PAGES 

50 

U,  MONITORING  AGENCY  NAME  & ADDRESS(7/  different  from  Controlling  Office) 

Defense  Advanced  Research  Projects  Agency  (ARP A) 
1400  Wilson  Boulevard 

15.  SECURITY  CLASS,  (of  this  report) 

Unclassified 

Arlington,  Virginia  22217 

1 5«.  DECLASSI  FI  CATION/ DOWN  GRADING 
SCHEDULE 

16.  DISTRIBUTION  STATEMENT  (of  thl a Report) 

Approved  for  public  release;  distribution  unlimited 

/ 

17.  DISTRIBUTION  STATEMENT  (ol  the  abstract  entered  In  Block  20,  If  different  from  Report) 

18.  SUPPLEMENTARY  NOTES 

19.  KEY  WORDS  ( Continue  on  reverse  aide  if  necessary  and  Identity  by  block  number) 

Electronic  Warfare  (EW)  suite 
Multi-attribute  utility  (MAU)  model 
System  evaluation 
Utility  assessment 
Decision  analysis 

20.  ABSTRACT  ( Continue  on  reverse  side  If  necessary  and  Identify  by  block  number) 

Introduction.  This  report  reflects  the  development,  validation,  and 
utilization  of  a multi-attribute  utility  (MAU)  model  for  use  by  the  Naval 
Electronic  Systems  Command  (NAVELEX)  in  comparing  and  evaluating  six  Electronic 
Warfare  (EW)  suite  design  proposals  submitted  by  contractors  under  new  design- 
to-cost  policies  enunciated  by  the  Department  of  Defense  (DoD)  in  its  directive 
5000.1  (Acquisition  of  Major  Defense  Systems)  and,  subsequently,  in  its 

directive  5000.28  (Design  to  Cost). 

DD  t j°nM73  1473  edition  of  i nov  65  is  obsolete  Unclassified 


SECURITY  CLASSIFICATION  OF  THIS  PACE  ( Whan  Data  Entatad) 

4 


41 


Unclassified 

SECURITY  CLASSIFICATION  OF  THIS  PAGEfHTiw  Bnt»nd) 


Background  and  Approach.  In  light  of  Secretary  of  the  Navy  In- 
struction (SECNAVINST)  5000.1,  which  implemented  DoD  Instruction  5000.1, 
NAVELEX  engaged  the  services  of  a contractor  (Decisions  and  Designs, 
Incorporated)  to  assist  in  the  development  of  a Request  for  Proposal 
(RFP)  and  a means  of  evaluating  the  design  proposals  which  would  be 
submitted  in  response  to  the  RFP  for  an  EW  system,  a group  of  sophisticated 
and  complex  suites  for  detecting  and  responding  to  different  threats  to 
different  Naval  surface  vessels  in  different  situations.  As  a result,  a 
MAU  model  was  developed  and  validated  for  use  by  NAVELEX. 

The  approach  used  in  the  development  of  the  MAU  model  was  to  create  a 
hierarchical  structure  by  means  of  which  higher- level  elements  were 
successively  decomposed  into  sub-elements  in  order  to  relate  the  degree 
to  which  measured  performance  levels  of  technical  system  characteristics 
enhanced  the  worth  of  the  EW  suite  design  proposals. 

After  completing  the  hierarchical  structure  of  the  model.  Navy  experts 
quantified  the  relationships  among  the  elements  at  the  same  levels, 
assessed  the  effectiveness  of  varying  performance  levels  of  controllable 
parameters  and  assigned  weights  to  each  component.  The  last  steps  in 
developing  the  model  consisted  of  validating  or  modifying  the  resulting 
model  in  the  light  of  sensitivity  analyses  and  calibration  based  on  data 
from  existing  EW  suites. 

Findings  and  Implications.  Implementation  of  DoD  Directive  5000.1  in 
the  design  and  evaluation  stage  of  the  Navy's  EW  System  marked  the  first 
time  that  the  design-to-cost  procurement  policies  had  been  implemented 
from  the  very  initiation  of  a program  to  procure  a major  military  system. 

It  also  marked  the  first  time  that  a MAU  model,  a device  with  broad 
decision  analytic  applications  to  problems  of  evaluation,  had  been 
applied  to  the  design  and  evaluation  of  a major  military  system.  In 
this  case,  several  iterations  of  the  model  reduced  the  number  of  contractors 
from  six  to  the  two  who  are  now  undertaking  engineering  development  of 
prototype  systems.  The  Navy  is  modifying  the  model  used  for  initial 
selection  in  order  to  evaluate  the  two  prototype  systems  and,  on  the 
basis  of  this  evaluation,  to  select  the  better  one. 

Although  the  development  and  utilization  of  the  MAU  model  was  successful 
in  the  case  of  the  Navy's  EW  System,  it  is  apparent  that  additional 
research  is  needed  to  (1)  ensure  the  systematic  selection  of  scenarios 
that  are  representative  of  the  population  of  possible  scenarios  and  that 
discriminate  among  the  alternatives  being  evaluated;  and  (2)  ascertain 
the  effects  of  approximating  complex  relationships  by  using  relatively 
simple  combination  rules. 
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SUMMARY 


Introduction.  This  report  describes  how  Department  of 
Defense  (DoD)  and  Secretary  of  the  Navy  (SECNAV)  design-to- 
cost  policies  led  to  the  development  and  use  of  a multi- 
attribute utility  (MAU)  model  for  evaluating  suite  design 
proposals  for  the  Navy's  Electronic  Warfare  (EW)  System.  It 
also  explains  how  that  model  or  evaluating  mechanism  functions. 

Background  and  Approach.  Faced  with  increasing  costs 
and  tightening  budgets  in  the  procurement  of  major  military 
systems,  DoD  promulgated  a new  procurement  policy  in  1971 
linking  designed  performance  and  its  attendant  benefits  to 
cost  ceilings.  This  policy,  which  is  enunciated  in  DoD 
Directive  5000.1,  stated  that: 

discrete  cost  elements  (e.g.,  unit  production  costs, 
operating  and  support  cost)  shall  be  translated  into 
'design  to'  requirements . ^ 

The  intent  of  this  policy  statement  was  to  emphasize  the 
need  to  balance  (by  means  of  an  interactive  design  process) 
system  performance,  costs,  and  schedules  with  the  quantities 
required  to  achieve  the  most  effective  military  capability 
with  available  DoD  resources. 

The  design-to-cost  concept  introduced  by  the  1971  DoD 
Directive  has  recently  been  superceded  by  DoD  Directive 
5000.28,  which  further  states  that: 

The  Design  to  Cost  concept  establishes  cost  as  a design 
parameter  during  a system's  design  and  development 
phase  and  provides  a cost  discipline  to  be  used  through- 
out the  acquisition  and  operation  of  a system. ^ 

Meanwhile,  in  light  of  Secretary  of  the  Navy  Instruction 
(SECNAVINST)  5000.1,  which  implemented  DoD  Instruction 
5000.1,  the  Naval  Electronic  Systems  Command  (NAVELEX) 
engaged  the  services  of  a contractor 3 to  assist  in  evalua- 
ting the  suite  design  proposals  which  would  be  submitted  in 


^Acquisition  of  Major  Defense  Systems,  DoD  Directive  5000.0, 
July  13,  1971 . 

2 

Design  to  Cost,  DoD  Directive  5000.28,  May  23,  1975. 

3 
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response  to  the  Request  for  Proposal  (RFP)  for  the  Navy's 
EW  System,  a group  of  sophisticated  and  complex  suites  for 
detecting  and  responding  to  different  threats  to  different 
naval  surface  vessels  in  different  situations. 

Results . Implementation  of  the  foregoing  policies  in 
the  design  and  evaluation  stages  of  the  Navy's  EW  System 
marked  the  first  time  that  design-to-cost  procurement 
policies  had  been  implemented  from  the  very  initiation  of  a 
program  to  procure  a major  military  system.  It  also  marked 
the  first  time  that  a MAU  model,  a device  with  broad  decision- 
analytic  applications  to  problems  of  evaluation,  had  been 
applied  to  the  design  and  evaluation  of  a major  military 
system.  In  this  case,  the  model  was  used  as  an  evaluating 
mechanism  to  select  two  contractors  from  among  six,  who  were 
then  to  initiate  the  engineering  development  of  prototype  EW 
suites . 

Research  Implications  for  MAU  Model  Development.  This 
application  pointed  to  two  important  methodological issues 
in  need  of  further  in-depth  research.  One  is  the  design  and 
use  of  scenarios  for  the  evaluation  of  suites  to  be  deployed 
in  an  uncertain  future.  The  other  is  the  sensitivity  of  MAU 
models  to  the  effects  of  approximating  the  complex  relation- 
ships that  exist  among  factors  in  the  model. 
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AN  APPLICATION  OF  MULT I -ATTRIBUTE  UTILITY  THEORY: 
DESIGN-TO-COST  EVALUATION  OF  THE  U.S.  NAVY'S 
ELECTRONIC  WARFARE  SYSTEM 


1.0  INTRODUCTION 


Faced  with  increasing  costs  and  tightening  budgets  in 
the  procurement  of  major  military  systems,  the  Department  of 
Defense  (DoD)  promulgated  a new  procurement  policy  in  1971 
linking  designed  performance  and  its  attendant  benefits  to 
cost  ceilings.  This  policy,  which  was  enunciated  in  DoD 
Directive  5000.1  and  subsequently  implemented  by  means  of 
instructions  published  by  each  of  the  Military  Services, 
stated  that: 

discrete  cost  elements  (e.g.,  unit  production  costs, 
operating  and  support  cost)  shall  be  translated  into 
'design  to'  requirements.  System  development  shall  be 
continuously  evaluated  against  these  requirements  with 
the  same  rigor  as  that  applied  to  technical  require- 
ments. Practical  tradeoffs  shall  be  made  between 
system  capability,  cost,  and  schedule.  Traceability  of 
estimates  and  costing  factors,  including  those  for 
economic  escalation,  shall  be  maintained.! 

The  intent  of  this  policy  statement  was  to  emphasize  the 
need  to  balance  (by  means  of  an  interactive  design  process) 
military  costs  and  management  with  the  quantities  required 
to  achieve  the  most  effective  military  capability  with 
available  DoD  resources. 

The  design-to-cost  concept  introduced  by  DoD  Directive 
5000.1  has  recently  been  superceded  by  DoD  Directive  5000.28, 
which  further  states  that: 

The  Design  to  Cost  concept  establishes  cost  as  a design 
parameter  during  a system's  design  and  development 
phase  and  provides  a cost  discipline  to  be  used  through- 
out the  acquisition  and  operation  of  a system. 2 


^"Acquisition  of  Major  Defense  Systems,  DoD  Directive  5000.1, 
July  IT,  lg'Tf:  
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Design  to  Cost,  DoD  Directive  5000.28,  May  23,  1975 
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Meanwhile,  in  light  of  Secretary  of  the  Navy  Instruction 
(SECNAVINST)  5000.1,  which  implemented  DoD  Instruction 
5000.1,  the  Naval  Electronic  Systems  Command  (NAVELEX) 
engaged  the  services  of  a contractor  to  assist  in  evaluating 
the  suite  design  proposals  which  would  be  submitted  in  re- 
sponse to  the  Request  for  Proposal  (RFP)  for  the  Electronic 
Warfare  (EW)  System,  a group  of  sophisticated  and  complex 
semi-automated,  computerized  suites  for  detecting  and 
responding  to  different  threats  to  different  naval  surface 
vessels  in  different  situations. 

1. 1 Purposes  of  the  Report 

The  purposes  of  this  report  are: 

1.  To  describe  how  the  effective  DoD  and  Secretary  of 
the  Navy  (SECNAV)  design-to-cost  policies  affect 
the  evaluation  phase  in  the  procurement  policy; 

2.  To  describe  and  compare  approaches  to  evaluation 
and  to  explain  the  selection  of  the  multi-attribute 
utility  (MAU)  approach; 

3.  To  relate  the  circumstances  in  which  the  Navy 
employed  the  MAU  approach  in  evaluating  EW  suite 
design  proposals; 

4.  To  explain  the  development  and  application  of  the 
MAU  model  for  evaluating  those  proposals;  and 

5.  To  outline  those  areas  in  which  further  research 
in  MAU  models  and  their  application  is  required. 

1 . 2 Background  and  Scope 

The  research  conducted  by  the  contractor  in  developing 
the  MAU  model  for  evaluating  EW  suite  design  proposals  has  a 
certain  historical  and  technical  interest.  Although  design- 
to-cost  policies  have  been  in  effect  since  1971,  their  early 
implementation  was  limited  to  systems  already  beyond  the 
design  and  development  stage  and  ensured  only  that  the 
systems  would  be  produced  within  previously  budgeted  limits. 
The  design  and  evaluation  stage  of  the  Navy's  EW  System, 
however,  marked  the  first  time  that  design-to-cost  procure- 
ment policies  had  been  implemented  from  the  very  initiation 
of  a program  to  procure  a major  military  system.  It  also 
marked  the  first  time  that  a MAU  model,  a device  with  broad 
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Decisions  and  Designs,  Incorporated  (DDI) . 
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decision-analytic  applications  to  problems  of  evaluation, 
had  been  employed  in  the  procurement  of  a major  military 
system.  Since  the  EW  System  is  not  yet  in  production,  much 
less  in  operation,  experience  with  this  and  other  systems 
procured  under  design-to-cost  policies  must  be  acquired 
before  a final  verdict  on  their  ultimate  value  and  that  of 
the  models  may  be  made.  This  report  makes  no  effort,  there- 
fore, to  address  this  issue. 

Instead,  it  attempts  only  to  set  forth  the  implications 
of  these  policies  for  design  and  to  show  how  a MAU  model  was 
employed  to  assist  in  evaluating  suite  design  proposals 
submitted  under  provisions  of  these  policies  and  served  as 
one  means  of  selecting  two  from  among  six  design  proposals 
for  further  engineering  development  of  prototypes  for 
testing.  Specifically,  although  a MAU  model  was  employed 
throughout  the  entire  evaluation,  it  was  developed  and 
applied  extensively  only  in  the  evaluation  of  technical 
systems  utility  (TSU) . Other  elements  of  military  worth  or 
of  cost  and  management  were  evaluated  differently. 
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2.0  DESIGN-TO-COST  PROCUREMENT  POLICIES  AND  THEIR 
IMPLICATIONS  FOR  TECHNICAL  SYSTEMS  DESIGN 


Changes  in  procurement  policies  obviously  involve  new 
procurement  practices  and  require  new  responses  in  system 
design.  Design-to-performance  and  design-to-cost  procure- 
ment policies  both  relate  performance  and  cost,  but  each 
does  so  differently,  by  placing  different  emphasis  upon  the 
one  or  the  other.  Since  their  impact  upon  system  design  is 
also  different,  an  understanding  of  the  differences  between 
the  two  kinds  of  procurement  policies  as  they  affect  system 
design  encompasses  the  reasons  for  the  change  and  suggests 
means  of  adapting  to  it. 

Under  a design-to-performance  procurement  policy, 
performance  requirements  in  the  RFP  are  stated  as  absolute 
descriptions  of  needs.  Any  system  failing  to  meet  the 
stipulated  requirements  is  defined  as  essentially  worthless. 
If,  for  example,  the  requirement  that  a military  system 
respond  within  ten  minutes  to  a distant  threat  were  taken 
literally,  a system  responding  within  ten-and-a-half  minutes 
would  be  regarded  as  worthless.  However,  in  most  cases, 
such  a system  would  not  be  worthless,  only  worth  less  than  a 
system  actually  satisfying  the  requirement.  Taken  lit- 
erally, such  a requirement  implies  a price-is-no-object 
attitude  toward  design  because  the  technology  necessary  to 
satisfy  the  requirement  may  be  extremely  expensive.  Al- 
though seldom  reflected  in  practice,  such  an  attitude  is 
implicit  in  performance  requirements  and  can  lead  to  over- 
designed  systems.  Since  over-design  is  as  much  an  economic 
as  an  engineering  concept,  a design-to-performance  procure- 
ment policy  may  over-emphasize  performance  and  lead  to  cost 
overruns,  reduced  acquisitions,  or  compensatory  sacrifices 
in  operational  performance  elsewhere  in  the  system. 

Under  a design-to-cost  procurement  policy,  these  prob- 
lems may  be  reduced  or  eliminated.  Rather  than  stating 
absolute  performance  requirements,  the  policy  stipulates  a 
production  cost  ceiling  for  each  unit  of  the  system  and 
allows  the  levels  of  performance  to  vary  within  certain 
limits.  These  variable  levels  of  performance  are  suitably 
valued  across  that  range.  Accordingly,  since  the  design  of 
a system  requires  the  maximum  value  of  performance  within  a 
budgetary  constraint,  over-design  is  less  likely,  military 
value  is  maximized,  and  the  risks  of  cost  overruns,  in- 
sufficient acquisitions,  and  compensatory  sacrifices  in 
operational  performance  are  minimized  in  most  cases.  There 
is  some  controversy  whether  or  not  more  modest  levels  of 
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performance  dictated  by  cost-per-unit  ceilings  will  be 
sufficient  to  meet  mission  requirements  in  certain  special 
cases.  This,  however,  is  not  an  issue  addressed  in  this 
report. 

For  this  contractor,  the  most  immediate  implications 
involve  his  responsibilities  for  a new  system.  Previously, 
when  a Military  Service  defined  the  technical  requirements 
of  a system,  the  contractor's  primary  responsibility, 
theoretically,  was  to  propose  a system  design  meeting  those 
requirements.  He  also  attempted  to  determine  a means  of 
producing  the  proposed  system  within  minimum  production 
standards  of  quality  at  a cost  lower  than  that  of  a com- 
petitor. Presently,  since  a Military  Service  simply  out- 
lines the  mission  requirements  of  a system  in  rather  general 
terms  and  states  the  production  cost-per-unit  ceiling  and 
since  all  contractors  have  the  same  production  ceilings  and 
standards  to  meet,  their  primary  responsibility  is  to  propose 
a system  design  offering  more  and  efficient  technical  ad- 
vantages, not  lower  cost,  than  does  a system  design  developed 
by  a competitor. 

In  practice,  the  design-to-cost  procurement  policy  is 
governed  by  three  dominant  considerations:  one,  the  prob- 

ability of  meeting  production  and  installation  schedules; 
two,  the  acceptability  of  development  and  acquisition  costs; 
and,  three,  military  worth. 

First,  after  issuing  an  RFP  and  receiving  proposals  in 
reply,  the  Military  Service  must  estimate  the  likelihood 
that  each  contractor  can  meet  the  production  and  instal- 
lation schedules  if  his  proposal  were  accepted.  This 
estimate  is  based  upon  assessments  of,  among  other  con- 
siderations, the  contractor's  past  performance  in  fulfilling 
Government  contracts  and  his  capabilities  to  produce  the 
proposed  system  along  with  his  other,  possibly  unrelated, 
production  commitments. 

Second,  the  Military  Service  must  judge  the  expense 
required  to  develop  a working  prototype  for  testing.  Since 
the  burden  of  designing  a system  under  design-to-cost  pro- 
curement policies  falls  largely  upon  each  contractor,  the 
Military  Service  has  paid  selected  contractors  to  develop 
their  proposals. 

Third,  (and  of  major  concern  in  this  case  study) , the 
Military  Service  must  evaluate  the  military  worth  of  a 
contractor's  design  proposal  and  compare  that  proposal  with 
other,  often  quite  different,  proposals.  For  implicit  in 
design-to-cost  procurement  policies  is  a tacit  reliance  upon 
the  ingenuity  of  each  contractor  to  achieve  maximum  value 
for  a stipulated  cost.  The  flexibility  permitted  by  the 
absence  of  specific  technical  requirements  allows  each 


5 


contractor  to  develop  a proposal  which  appears  to  him  to 
meet  best  the  requirements  of  a given  system.  Since  flexi- 
bility allows  diversity — each  contractor  develops  a design 
that  reflects  his  particular  corporate  interests,  his  under- 
standing of  the  mission,  and  his  technical  and  manufacturing 
capabilities — some  method  of  evaluating  system  design  proposals 
that  may  be  quite  different  from  each  other  is  required. 

The  general  requirements  for  such  an  evaluating  mechanism 
are  described  in  Section  3.0. 
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3.0  GENERAL  APPROACHES  TO  TECHNICAL  SYSTEMS  EVALUATION 


3 . 1 Criteria  for  Evaluating  Mechanisms 

The  logic  of  evaluation  and  the  nature  of  competitive 
bidding  require  that  any  program  for  evaluating  design 
proposals  be  carefully  constructed  to  meet  four  general 
criteria;  namely,  it  must: 

1.  Discriminate  effectively  among  the  alternative 
design  proposals.  Any  program  of  evaluation  must 
not  only  differentiate  among  proposed  designs,  but 
also  rank  them  in  order  of  their  worth. 

2.  Be  reliable.  The  program  must  reproduce  for  any 
evaluator  consistently  similar  results  from  sim- 
ilar information  and  options. 

3.  Be  intelligible.  The  program  must  have  an  explicit 
logic  that  facilitates  understanding  of  the  relation- 
ships between  the  data  and  the  results  of  the 
evaluation. 

4.  Be  equitable.  The  program  must  have  no  inherent 
bias.  Each  design  proposal  must  have  the  same 
chance  a priori  to  be  selected  as  does  any  other 
proposal. 

These  criteria  are  fairly  general  and  do  not  preclude 
the  use  of  a number  of  different  procedures  or  approaches, 
like  simulation  runs,  global  assessments,  or  MAU  models.  In 
fact,  it  is  common  practice  for  two  or  more  of  these  pro- 
cedures or  approaches  to  be  employed  simultaneously  on 
different  aspects  of  an  evaluation  problem.  Any  program  for 
evaluation,  however  constituted  but  meeting  these  criteria, 
has  but  one  operational  purpose:  to  relate  available  data 

relevant  to  the  evaluation  of  a proposed  system  design 
to  its  total  worth  to  the  Military  Service.  Since  the  scope 
of  this  report  is  limited  to  the  use  of  a MAU  model  in  evalua- 
ting technical  systems  utility  (TSU)  of  the  EW  suite  design 
proposals,  the  purpose  of  this  particular  model  is  to  relate 
measures  of  performance  levels  of  the  controllable  parameters 
of  a design  proposal  to  the  military  worth  of  the  proposed 
system.  Examples  of  such  parameters  are  detection  range, 
number  of  targets  prior  to  saturation,  and  effective  radiated 
power.  For  the  sake  of  brevity,  "performance  levels  of  the 
controllable  parameters"  will  be  called,  simply,  "technical 
characteristics . " 

The  first  two  of  these  criteria,  discrimination  and 
reliability,  are  satisfied  or  not  by  the  nature  of  the 
model — in  this  case,  the  MAU  model — for  evaluating  design 
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proposals.  A means  of  ensuring  that  the  third  criterion, 
intelligibility,  is  fulfilled  is  to  structure  the  evaluating 
mechanism  so  that  the  paths  from  the  data  to  the  results  of 
the  evaluation  are  explicit  in  the  model. 

To  address  the  fourth  criterion,  equity,  an  evaluating 
mechanism  for  relating  the  data  derived  from  diverse  design 
proposals  to  the  results  of  the  evaluation  must  be  compre- 
hensive and  flexible.  Comprehensiveness  ensures  that  all 
possible  considerations  known  to  be  relevant  prior  to  eval- 
uating a design  proposal  are  encompassed  by  the  evaluating 
mechanism.  Flexibility  allows  modification  of  the  evaluating 
mechanism  in  the  event  of  unforeseen  relevant  factors  and 
thus  ensures  that  arbitrary  stipulations  do  not  bias  the 
evaluating  mechanism  against  distinctive  design  proposals 
allowed  by  the  latitude  of  design-to-cost  RPF's.  An  eval- 
uating mechanism  meeting  these  four  criteria  ensures  equity 
because  it  enables  the  Military  Service  to  justify  its 
selection  or  allows  contractors  to  challenge  it  on  a rational 
and  constructive  basis. 

3 . 2 Possible  Methodologies  for  Technical  Systems  Evaluation 

The  two  questions  addressed  by  a methodology  for 
evaluating  TSU  of  design  proposals  which  meets  the  four 
criteria  are:  What  is  the  actual  performance  of  a system 

built  according  to  its  design  proposal  likely  to  be,  and 
what  is  that  level  of  actual  performance  worth?  The  first 
question  requires  a prediction  as  an  answer,  the  second  an 
evaluation.  To  predict  performance  of  proposed  systems,  it 
would  be  ideal  to  have  a clairvoyant  capable  of  looking  into 
the  future  in  which  the  systems  were  built  and  of  observing 
their  technical  effectiveness.  Since  no  such  clairvoyant 
exists,  the  problem  becomes  one  of  making  the  best-informed 
and  best-reasoned  predictions  possible.  To  evaluate  the 
predicted  performance  of  such  systems,  it  would  be  ideal  to 
have  a permanent  and  absolute  standard  of  value.  In  lieu  of 
one,  the  problem  becomes  one  of  making  the  best  judgments 
possible. 

The  possible  methodologies  for  system  evaluation  all 
involve  one  way  o another  both  questions.  It  should  be 
noticed  that  questions  requiring  a prediction  should  be 
answered  separately  from  questions  requiring  an  evaluation. 
Answers  to  questions  requiring  a prediction  concern  the 
effectiveness  of  the  proposed  system;  answers  to  questions 
requiring  an  evaluation  concern  the  utility  of  the  proposed 
system.  These  distinctions  between  prediction  and  eval- 
uation, and  between  effectiveness  and  utility  will  be 
observed  throughout  the  rest  of  this  paper  as  the  details  of 
possible  methodologies  are  discussed.  However,  the  problem 
to  which  such  methodologies  address  themselves  will  be 
regarded  as  one  of  evaluation  in  a broader  sense. 
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3.2.1  Approaches  to  predictions  - There  are  essentially 

two  kinds  of  approaches  to  predictions:  1)  personal  assess- 

ments ranging  from  guesses  or  hunches  to  expert  opinion;  and 
2)  predictive  models.  The  line  between  the  two  is  not  so 
sharp  that  models  may  not  be  biased  in  one  way  or  another 
and  is  further  blurred  by  vagueness  about  the  point  at  which 
a proposal  becomes  too  complex  for  a simple  personal  assess- 
ment. Since  most  evaluation  problems  in  the  procurement  of 
military  systems  are  very  complex,  however,  predictive 
models,  because  they  pool  knowledge  and  sophistication  of 
many  experts,  are  preferred  sources  of  predictions.  Of 
these,  simulation  models  are  commonly  the  most  effective 
means  of  predicting  the  effectiveness  of  proposed  systems. 

The  impressiveness  of  simulation  runs  in  predicting  per- 
formances, however,  is  no  substitute  for  evaluating  them. 
Since  prediction  is  quite  different  from  evaluation,  no 
amount  of  performance  data  can  yield  a judgment  of  even  so 
simple  a kind  as  "more  is  better"  without  involving  a pre- 
scriptive, not  a descriptive,  assertion.  The  point  needs  to 
be  emphasized  because  simulation  runs,  while  often  desirable 
in  any  evaluation  of  system  design  proposals,  are  not  by 
themselves  a possible  methodology  for  system  evaluation  in 
the  broad  sense  of  the  term;  rather,  they  are  a means  of 
making  predictions  of  performance  characteristics. 

3.2.2  Approaches  to  evaluations  - There  are 

essentially  two  kinds  of  approaches  to  evaluations:  1) 

global  and  2)  decomposed.  The  line  between  the  two  is  not 
always  a sharp  one  because  most  global  judgments  result  from 
a few  judgments  of  lesser  scope.  The  difference  between 
them,  though  theoretically  one  of  degree,  is  sharp  enough, 
however,  to  separate  those  approaches  that  are  general  and 
implicit  and  those  that  are  specific  and  explicit. 

A global  approach  to  evaluation  consists  of 
global  judgments  which  are  ratings  of  an  entire  system  (or 
an  entire  sub-system)  on  one  or  more  scales  and  are  often 
accompanied  by  written  explanations.  Because  the  scope  of 
global  judgments  is  very  large,  there  is  usually  a broad 
intuitive  leap  from  the  information  at  hand  to  the  judgment 
in  mind.  Nevertheless,  in  evaluating  systems  or  sub-systems 
that  are  not  complex,  a global  approach  may  be  entirely 
adequate  and  appropriate. 

The  difficulties  of  the  global  approach  arise 
most  noticeably  in  cases  involving  the  evaluation  of  complex 
proposals.  These  include  the  inability  to  use  all  the 
information  available,  the  instability  of  evaluating  stan- 
dards, and  a variety  of  biases.  As  a general  rule,  the  more 
heterogeneous  and  complex  the  information,  the  lower  the 
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quality  of  the  global  judgment.  Aids  for  reducing  (or  elimi- 
nating) the  impact  of  these  difficulties  have  been  suggested 
but  have  not  as  yet  proven  themselves  effective.! 

A global  approach  to  evaluation  of  complex 
design  proposals,  then,  is  not  presently  bound  to  satisfy  at 
least  three  of  the  four  criteria  for  an  evaluating  mechanism. 
Although  it  can  discriminate  among  design  proposals,  it 
cannot  ensure  its  reliability  or  intelligibility.  Accordingly, 
there  can  be  no  assurance  of  its  equity.  These  difficulties 
ultimately  result  from  the  broad  intuitive  leap  from  infor- 
mation to  judgment. 

The  decomposed  approach,  on  the  contrary,  is  a 
means  of  systematically  combining  information  and  specific 
judgments  to  produce  an  evaulation  of  a design  proposal. 

One  such  approach  is  known  as  a multi-attribute  utility 
approach;  its  central  principle  is  to  address  an  evaluation 
problem  by  decomposing  all  the  elements  relevant  to  it  into 
the  smallest  possible  sub-elements  and  then  to  combine  them, 
first  into  increasingly  aggregated  measures  of  effectiveness 
based  upon  predicted  performance  characteristics  and  then 
into  increasingly  aggregated  estimates  of  utilities. 

Since  one  of  the  purposes  of  this  report  is  to 
explain  the  development  and  application  of  the  MAU  model  for 
evaluating  EW  suite  design  proposals,  especially  their  TSU,’ 
the  explanation  in  Section  5.0  will  show  the  ways  in  which 
the  model  satisfied  the  four  criteria  of  evaluating  mechanisms. 


See,  for  example,  Paul  Slovic  and  Sarah  Lichtenstein,  "Comparison 
of  Bayesian  and  Regression  Approaches  to  the  Study  of  Infor- 
mation Processing  in  Judgment,"  Organizational  Behavior  and 
Human  Performance,  6.6  (November,  1971),  649-744. 
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4.0  IMPLEMENTING  THE  DESIGN-TO-COST  PROCUREMENT  POLICIES 
FOR  THE  NAVY'S  ELECTRONIC  WARFARE  SYSTEM 


4 . 1 Developing  the  Request  for  Proposal 

Design-to-cost  procurement  policies,  since  they  require 
that  design  be  integrated  with  management  and  cost  in  the 
procurement  process,  imply  the  need  for  developing  an  RFP 
with  regard  to  the  nature  of  the  responsive  design  proposals. 
The  process  begins  with  Navy  experts  addressing  two  general 
kinds  of  questions:  one,  what  kind  of  performance  is  re- 

quired to  accomplish  what  kinds  of  missions  against  what 
kinds  of  threats  in  what  kinds  of  environments;  and  two, 
what  is  such  performance  worth  and  what  factors  affecting 
operational  performance  can  enhance  or  compensate  for  it? 

The  RFP  which  addressed  these  kinds  of  questions  avoided 
answering  them  with  system  requirements.  Instead,  it  called 
for  the  submission  of  design  proposals  for  five  interrelated 
suites  comprising  the  EW  System — each  suite  with  its  own 
cost-per-unit  ceiling  and  each  for  a different  class  of 
ships — and  outlined  the  missions  and  scenarios  for  their 
deployment.  The  RFP  also  indicated  those  considerations  of 
particular  importance  to  the  Navy,  modularity  and  growth. 

In  providing  funds  to  contractors  for  preparing  their  EW 
suite  design  proposals,  the  Navy  reduced  expense  while  main- 
taining competition  by  initially  selecting  six  from  among  those 
responding  as  most  capable  of  designing  and  producing  an  EW 
system. 

4.2  The  Evaluation  Problem 


Since  the  design-to-cost  procurement  policy  permits, 
even  encourages,  diversity  in  design  proposals,  and  since 
the  RFP  for  an  EW  suite  was  general  enough  to  ensure  such 
diversity,  the  Navy  faced  the  problem  of  evaluating  six 
feasible,  but  potentially  quite  different,  design  proposals 
and  reducing  this  number  to  two.  The  two  finalists  were 
then  to  undertake  engineering  development  of  prototypes  for 
testing  and  to  submit  more  specific  proposals  for  meeting 
management  and  cost  objectives  from  production  through 
operational  deployment. 

The  general  evaluation  problem  was,  of  course,  to 
evaluate  the  contractors'  proposals  in  their  entirety,  that 
is,  their  military  worth,  cost,  and  management.  Although 
the  decomposed  approach  could  have  been  used  for  cost  (life- 
cycle  cost  and  risk  with  estimated  cost)  and  management 
(organization,  plans  and  schedules,  review  cycles,  change 
mechanisms,  and  the  like) , it  was  not.  Nor  were  sub-elements 
of  each  system  comprising  military  worth  (acceptability  of 
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operational  performance)  similarly  decomposed.  In  .fact,  of 
eight  such  sub-elements,  only  TSU  was  decomposed  into  a 
hierarchically-structured  model.  The  modeling  effort  de- 
scribed henceforward  in  this  report  concerns  the  determination 
only  of  TSU;  that  is,  the  Navy  employed  a fjilly  developed 
MAU  model  in  evaluating  the  benefit  derived  only  from  tech- 
nical performance  of  design  proposals.  To  address  this 
specific  evaluation  problem,  Navy  experts,  assisted  by 
analysts  of  the  supporting  contractor^,  developed  a MAU 
model,  or  evaluating  mechanism,  that  would  satisfy  the 
criteria  for  evaluation. 

4 . 3 The  Methodology  Selected 

To  evaluate  TSU  of  the  EW  suite  design  proposals,  the 
Navy  preferred  a model  combining  a decomposed  approach  sup- 
ported by  simulation  runs  to  a global  approach  supported  by 
them.  These  two  approaches  are  represented  graphically  in 
Figure  4-1. 
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TWO  POSSIBLE  METHODOLOGIES  FOR  COMPLEX  EVALUATIONS 
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Decisions  and  Designs,  Incorporated  (DDI) . 
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One  of  the  essential  differences  between  them  is  that  the 
global  approach  more  extensively  employs  simulation  runs 
than  does  the  decomposed  approach.  The  use  of  simulation 
runs  in  the  global  approach  extends  up  through  certain 
contextual  aspects  of  performance,  whereas  in  the  decomposed 
approach  it  extends  no  further  than  the  strictly  technical 
aspects  of  performance.  Another  difference,  of  course,  is 
that  the  global  approach  attempts  to  evaluate  military  worth 
by  means  of  global  judgments,  and  the  decomposed  approach, 
in  this  case,  by  means  of  a formal  MAU  model. 

The  Navy  preferred  the  latter  approach  for  basically 
two  reasons.  It  found  that  the  level  of  effort  necessary  to 
use  simulation  runs  at  higher  levels  would  result  in  a cost 
exceeding  funds  allotted  for  evaluation.  And  it  anticipated 
that  the  decomposed--henceforth,  MAU — approach  would  be  more 
intelligible  and  its  requests  more  justifiable  than  the 
g loba 1 approach . 2 

In  developing  the  model  for  TSU  Navy  experts,  on  the 
basis  of  analytical  studies  and  their  operational  experience, 
assessed  the  probabilities,  weights,  and  utility  curves 
called  for  by  the  model.  In  particular,  the  structure  and 
nature  of  the  MAU  model  translated  the  data  reflecting 
different  levels  of  performances  of  different  kinds  into  a 
unified  figure  representing  the  total  technical  systems 
utility  of  each  EW  suite  proposed.  In  doing  so,  the  MAU 
model  attempted  to  evaluate  in  detail  the  TSU  of  each  of  the 
six  suite  design  proposals  without  prejudice  toward  any  of 
them  for  using  new  or  different  technologies. 

In  applying  this  model,  the  Navy  appraised  the  figures 
of  performance  stated  or  implied  in  each  EW  suite  design  pro- 
posal, for  the  Navy  reserved  to  itself  and  exercised  the  right  to 
assess  whether  or  not  the  figures  were  reasonable.  If  they 
were  considered  to  be  reasonable,  they  were  incorporated 
directly  into  the  evaluating  mechanism;  if  not,  they  were 
amended  by  the  Navy  on  the  basis  of  its  analysis  of  the 
proposed  design  and  then  incorporated  into  the  evaluating 
mechanism.  From  among  the  original  six  contractors  sub- 
mitting design  proposals,  the  evaluating  mechanism  served  as 
one  basis  for  selecting  two  to  proceed  with  engineering 
development  of  prototypes  for  testing.  The  ranking  of  all 


2 

The  difference  between  static  and  dynamic  models  was  not 
perceived  to  affect  the  appropriateness  of  using  such  a 
model  for  a dynamic  process.  Although  MAU  models  are  usually 
of  a static  nature,  they  can  be  used  to  make  reasonable 
approximations  of  dynamic  processes  under  special  circumstances . 
These  occur  in  the  case  of  EW  suites  since  their  weapon 
delivery  times  are  almost  instantaneous. 
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six  suite  design  proposals  according  to  TSU  was,  in  the  final 
decision,  slightly  affected  by  other  considerations  bearing 
upon  military  worth  and  by  consideration  of  management  and 
cost  in  the  final  evaluation. 

In  short,  the  MAU  model,  or  evaluating  mechanism, 
performed  in  accordance  with  expectations.  It  discriminated 
among  the  proposals  submitted  and  selected  two  finalists. 

And  it  did  so  fairly  since  the  mechanism  showed  no  bias  for 
either  innovative  or  conservative  proposals.  In  fact,  the 
proposals  which  were  selected  for  engineering  development 
and  testing  were  quite  different  from  each  other. 
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5.0  DEVELOPMENT  OF  THE  MULTI-ATTRIBUTE 
UTILITY  MODEL  FOR  DETERMINING 
TECHNICAL  SYSTEMS  UTILITY 


The  central  problem  in  any  effort  to  evaluate  military 
system  design  proposals  for  the  purpose  of  selecting  a con- 
tractor to  build  the  system  is  to  relate  its  predicted 
technical  performance  to  military  worth.  The  evaluating 
approach  is  the  means  of  relating  predicted  performance  to 
military  worth.  Under  design-to-cost  procurement  policies, 
the  evaluating  mechanism  must  be  capable  of  discriminating 
the  relative  military  worths  of  different,  sophisticated, 
and  complex  design  proposals  reliably,  intelligibly,  and 
fairly,  without  penalizing  innovative  proposals  submitted 
under  those  policies.  In  the  case  of  the  Navy's  EW  System, 
the  evaluating  mechanism  developed  was  a MAU  model  which 
translated  predicted  levels  of  performance  of  controllable 
parameters,  or  performance  characteristics,  into  a unified 
estimate  of  TSU  for  each  of  the  EW  suite  design  proposals. 

The  essential  fact  about  decomposed  models,  especially 
MAU  models,  is  that  they  are  explicitly  structured  to  manage 
information  so  that  one  or  more  experts  may  contribute  to 
the  final  evaluation  in  a logical  manner.  The  structural 
principle  of  the  MAU  model  used  in  this  case  was  hierar- 
chical decomposition;  that  is,  each  element  of  the  par- 
ticular evaluation  problem  was  analyzed  into  its  sub-elements, 
each  of  which  in  turn  analyzed  into  its  respective  sub- 
elements until  the  most  minute  sub-elements  of  a controllable 
parameter  of  performance  demonstrably  affecting  performance 
were  reached.  Conversely,  predicted  levels  of  performance 
of  the  lowest-level  sub-elements  in  the  model  may  be  aggre- 
gated until  a unique  figure  representing  TSU  is  determined. 
Accordingly,  expert  opinions  may  be  selectively  involved  for 
the  appropriate  component  or  sub-element. 

In  the  discussion  which  follows,  the  structure  of  the 

MAU  model  developed  as  the  evaluating  mechanism  for  EW  suite 

design  proposals  is  described;  that  part  of  the  model  used 
for  evaluating  TSU  is  then  explained.  In  this  second  part 
of  the  discussion,  terminology  will  be  defined  precisely  and 
the  rationale  for  this  part  of  the  model  will  be  offered. 

5 . 1 Description  of  the  Model  for  Evaluating  Design  Proposals 

The  hierarchical  structure  of  the  MAU  Model  consisted 
of  nine  levels,  six  of  which  are  illustrated  in  Figure  5-1, 

the  remaining  three  of  which  are  illustrated  in  Figure  5-2. 
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Figure  5-1 

THE  STRUCTURE  OF  THE  MULTI-ATTRIBUTE  UTILITY 
MODEL  FOR  ELECTRONIC  WARFARE  SYSTEM  PROPOSALS 


The  first  level  of  the  hierarchical  structure  corresponds 
to  the  single  figure  representing  the  total  worth  of  each  EW 
Suite  design  proposal. 

The  second  level  represents  three  major  elements: 
military  worth,  cost,  and  management,  the  first  of  major 
concern  in  this  article. 

The  third  level  represents  the  conditioning  variables 
ship  platforms.  The  five  values — $300,000;  $500,000; 
$1,000,000;  $2,000,000;  and  $3,000,000 — represent  the  cost- 
per-unit  ceiling  of  the  five  EW  suites  constituting  the  entire 
EW  System,  each  cost  configuration  associated  exclusively 
with  a single  class  of  ship  platforms.  For  example,  the 
$300,000  suite  was  intended  for  destroyer  escorts,  the 
$500,000  cost  configuration  for  destroyers,  the  $2,000,000 
for  cruisers.  These  conditioning  variables  are  necessary  for 
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determining  the  importance  of  technical  characteristics  and 
aggregates  thereof.  The  evaluating  mechanism  uses  these 
variables  to  help  distinguish  the  importance  of  protecting 
one  kind  of  ship  rather  than  another . 1 

The  fourth  level  represents  eight  functional  or  deploy- 
ment aspect  of  a system. ^ 

The  fifth  level  represents  four  conditioning  variables, 
each  delineating  a different  scenario  describing  missions 
and  threats  to  the  completion  of  those  missions;  the  four 
scenarios  are  independent  ship,  open  sea,  underway  replenish- 
ment, and  amphibious.  Although  a scenario  has  no  value  in  it- 
self, the  model  indicates  the  probability  that  the  situations 
described  will  occur  and  the  relative  importance  of  threats 
to  the  missions. 

The  sixth  level  represents  four  kinds  of  responses  to 
a threat. 
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Figure  5-2 

THE  SUB  STRUCTURE  OF  THE  MULTI-ATTRIBUTE 
UTILITY  MODEL  FOR  THE  DECOY  IN  THE  OPEN  SEA  SCENARIO 


^As  it  turned  out,  the  evaluation  revealed  that  the  Navy  max- 
imized military  worth  by  developing  an  EW  system  consisting  of 
three,  rather  than  five  configurations.  It  specified  one 
$1, 400, 000-per-unit  suite  instead  of  the  three  suites  with 
cost-per-unit  ceilings  of  $1,000,000;  $2,000,000;  and  $3,000,000. 

2 

During  evaluation,  aircraft  compatibility  was  dropped  from 
consideration. 
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The  seventh,  eighth,  and  ninth  levels  are  illustrated 
in  Fiqure  5-2. 

The  seventh  level  represents  the  operational  aspects  of 
these  responses. 

The  eighth  and  ninth  levels  represent  the  parameters  of 
performance  associated  with  a particular  aspect  of  a response 
in  a particular  situation. 

Note  that  the  heavy  line  in  Figure  5-1  represents  but 
one  possible  thread  through  this  hierarchy  from  a response 
to  a threat  to  proposal  worth.  A similar  heavy  line  might 
have  been  drawn  in  Figure  5-2  from,  say,  frequency  coverage 
up  to  effector  effectiveness  and  up  to  decoy  effectiveness. 

Since  there  is  a large  number  of  possible  threads,  this 
hierarchical  structure  organizes  in  a manageable  way  the 
great  quantity  of  data  needed  to  determine  the  worth  of  alter- 
native, complex  EW  system  proposals. 

5. 2 Aspects  of  the  Model 

5.2.1  Importance  of  structuring  - Developing  a MAU 
model  is,  of  course , a complex  and  demanding  task.  The  many 
considerations  addressed  in  developing  the  model  are  finally 
organized  hierarchically;  more  general  elements  are  suc- 
cessively decomposed  until  a structure  is  developed  that 
aggregates  measures  of  performance  characteristics  up  through 
TSU  of  each  design  proposal  or  that  justifies  any  higher- 
level  figure  of  utility  or  effectiveness  by  reference  to 
lower- level  figures.  For  these  reasons,  correct  structuring 
of  the  problem  is  the  most  important  step  in  the  process. 
Moreover,  correct  structuring  helps  to  reduce  possible 
errors  in  judgment  and  to  ensure  that  variations  in  the 
design  proposals  are  reflected  in  different  figures  of  TSU. 

5.2.2  Definitions  of  "utility"  and  "effectiveness"  - 
The  words  "utility"  and  "effectiveness"  are  used  in  this 

report  in  senses  consistent  with,  though  not  necessarily  so 
specific  as,  their  usage  in  related  disciplines.  To  some 
extent,  the  words  are  interchangeable;  the  summary  measures 
at  any  particular  level  in  the  hierarchy  for  TSU  could  logically 
be  called  utility  measures  or  effectiveness  measures,  for 
both  terms  indicate  measures  which  translate  technical  char- 
acteristics or  aggregates  thereof  onto  a scale  of  benefit,  or 
utility.  Ordinary  usage,  however,  distinguishes  the  two 
terms.  "Utility"  is  the  more  general  term  since  it  is  appli- 
cable to  all  subjective  evaluations  of  worth  at  all  levels 
from  highest  to  lowest  in  a MAU  model.  Conversely,  "effec- 
tiveness" is  restricted  in  its  usage  to  levels  in  the  model 
up  to  but  not  beyond  that  involving  the  performance  of  the 
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mission.  As  a result,  conventional  usage  makes  it  possible 
to  speak  of  the  benefit,  or  utility,  of  being  in  one  situation 
to  being  in  another  but  not  to  speak  of  the  effectiveness  of 
being  in  one  situation  or  another.  Accordingly,  the  word 
"utility"  is  not  inevitably  constrained  to  usage  associated 
with  the  higher  levels  in  the  hierarchy  as  "effectiveness" 
is  constrained  to  usage  associated  with  lower  levels  in  the 
hierarchy.  There  should  be  no  suggestion,  however,  that 
utility  is  better  or  more  significant  than  effectiveness; 
one  is  mainly  concerned  with  evaluation  (in  the  narrow 
sense) , the  other  with  prediction,  and  each  has  its  im- 
portant part  in  evaluation  (in  the  broad  sense) . The 
distinction,  admittedly  slight  (see  Figure  4-1  for  the  area 
in  which  simulation  runs  may  or  may  not  be  used) , is  that 
between  determinations  essentially  involving  military 
trade-offs  and  those  involving  technical  system  capabilities. 

5.2.3  Scaling  utility  and  effectiveness  - Since  a MAU 
model  involves  many  different  kinds  of  elements  and  many 
different  units  of  measurement,  it  is  important  to  have  a 
common  unit  of  measurement  that  makes  all  measures  commen- 
surable to  facilitate  aggregation  up  through  the  model. 

Such  a unit  is  utility  or  effectiveness,  and  the  convention 
is  to  scale  either  one  from  zero  to  100.  Accordingly, 
various  performance  characteristics  may  be  measured  as  a 
percentage — of  what,  we  shall  consider  later.  Once  trans- 
lated into  percentages,  they  become  amenable  to  aggregation 
at  higher  levels  in  the  MAU  model.  Moreover,  since  the 
model  is  hierarchical,  the  manner  in  which  a change  in 
performance  characteristics  affects  aggregated  utility  or 
effectiveness  at  successively  higher  levels  can  readily  be 
traced  throughout  the  model  and  conveniently  adapted  for  use 
on  an  interactive  graphics  computer. 

5 . 3 Steps  in  Developing  the  Model 

The  MAU  model  is  developed  in  five  basic  steps:  1) 

decomposing  the  evaluation  problem  into  its  structured 
elements,  2)  defining  element  relationships,  3)  establishing 
element  boundaries,  4)  developing  effectiveness  curves,  and 
5)  determining  element  weights.  Although  these  steps  must 
be  listed  and  discussed  sequentially,  they  should  be  under- 
stood to  be  taken  iteratively  in  practice,  especially  the 
last  three. 


5.3.1  Decomposing  the  evaluation  problem  - The  first 
step  is  to  decompose  the  evaluation  problem  into  its  struc- 
tured elements.  The  principle  of  hierarchical  decomposition 
identifies  the  elements  as  it  structures  them,  as  any  ana- 
lytic process  relates  parts  to  wholes.  Errors  can,  of 
course,  be  corrected,  but  their  ratification  requires  a 
readjustment  in  all  subsequent  steps  involving  structuring 
related  elements. 
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There  are  two  general  kinds  of  elements  that 
emerge  in  developing  a MAU  model  of  TSU:  one,  the  con- 

trollable parameters  of  performance  and,  two,  the  factors 
mediating  between  these  parameters  and  the  aggregate  TSU  of 
the  EW  suite.  The  controllable  parameters  of  performance 
require  no  further  explanation;  they  are  the  technical 
characteristics  of  the  systems.  The  factors  mediating 
between  the  parameters  and  the  aggregate  TSU  do;  the  in- 
clusion and  nature  of  some  factors  in  the  MAU  model  is  not 
always  self-evident. 

All  mediating  factors  define  the  context  in 
which  the  utility  or  effectiveness  of  each  sub-factor  is 
assessed.  One  kind  of  mediating  factor  is  an  aggregate  of 
utility  or  effectiveness  which  summarizes  measures  of  utility 
or  effectiveness  of  performance  characteristics.  For  example, 
one  can  speak  of  the  effectiveness  of  a decoy  as  well  as  of 
the  effectiveness  of  its  sensor.  The  other  kind  of  medi- 
ating factor  is  called  a conditioning  variable,  a contextual 
construct  for  modeling  purposes.  For  example,  although  one 
can  speak  of  the  effectiveness  of  a decoy,  one  cannot  speak 
of  the  effectiveness  of  an  open  sea  scenario.  The  difference 
is  that  between  those  factors  which  pertain  to  the  system 
itself  and  those  which  pertain  to  constraints  upon  its  cost 
or  to  the  situations  in  which  it  is  employed. 

Conditioning  variables  constitute  an  explicit 
recognition  of  the  fact  that  the  utility  or  effectiveness  of 
the  same  measure  of  performance  may  change  with  a change  in 
the  context  of  that  performance.  A conditioning  variable  is 
a set  of  conditions  that  provides  a context  in  which  measures 
of  performance  are  predicted  to  occur  and  are  evaluated  as  a 
percentage  of  maximum  utility  or  effectiveness.  In  the 
present  MAU  model  of  TSU,  there  is  only  one  level  of  con- 
ditioning variables.  It  represents  selected  military  envi- 
ronments in  which  an  EW  suite  may  be  operationally  deployed. 
These  environments  in  which  all  the  events  describing  the 
contexts  of  an  EW  suite  operationally  deployed  are  called 
scenarios.  Since  the  MAU  model  was  used  to  determine  TSU  of 
each  of  five  suites  defined  by  their  cost  configuration 
(cost-per-unit  ceiling)  and  employed  four  scenarios  desig- 
nating four  different  situations,  the  model  yields  20  different 
measures  of  TSU  for  each  EW  system  proposal.  For  example,  at 
the  same  level  of  technical  performance,  the  effectiveness 
of  the  decoy  in  the  $3,000,000  suite  in  the  open  sea  scenario 
might  be  far  greater  than  that  in  the  $500,000  configuration 
in  the  amphibious  scenario. 

The  use  of  scenarios  in  evaluation,  especially 
in  MAU  modeling,  is  necessary  as  an  economical  means  of 
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predicting  and  evaluating  performance.  It  is  also  necessary 
as  a means  of  reducing  to  a manageable  number  the  infinite 
number  of  details  required  to  describe  an  infinite  number  of 
possible  futures.  Scenarios  serve,  then,  as  classifying 
abstractions,  that  is,  sets  of  generalized  features  which 
characterize  a class  of  possible  futures. 

For  purposes  of  modeling,  even  this  greatly 
reduced  number  of  generalized  descriptions  is  neither 
economical  nor  manageable.  Accordingly,  a selection  of 
scenarios  must  be  made  to  ensure  that  the  sampling  is  rep- 
resentative of  what  the  future  may  be  like  on  the  one  hand 
and  that  the  selection  discriminates  among  the  design  pro- 
posals on  the  other.  Conversely,  scenarios  which  are  not 
representative  cannot  help  evaluate  proposed  designs  in  the 
situations  for  which  they  are  intended,  and  those  which  do 
not  discriminate  among  the  proposals  do  not  address  them- 
selves to  one  of  the  principal  criteria  for  evaluation.  The 
procedures  for  selecting  an  economical  and  manageable  number 
of  scenarios  that  are  at  once  representative  and  discrim- 
inating raise  difficult  methodological  questions. 3 Never- 
theless, they  have  been  sufficiently  developed  and  refined 
by  experience  to  enable  the  selection  of  scenarios  that  can 
serve  as  a basis  for  dependable  evaluation. 4 

5.3.2  Quantifying  the  model  - The  remaining  four 
steps — defining  the  element  relationships , establishing 
element  boundaries,  developing  effectiveness  curves,  and 
determining  element  weights — involve  a highly  iterative 
process.  It  goes  on  continuously  and  leads  to  refine- 
ments of  initial  quantifications  until  the  model  is  satis- 
factory to  those  who  must  use  it.  Since  the  model  must 
generate  figures  indicative  of  the  relative  TSU  of  design 
proposals,  the  relationships  among  the  elements  as  well  as 
their  measures  of  utility  and  effectiveness  must  be  ex- 
pressed mathematically. 

One  of  the  most  important  steps  in  developing 
the  MAU  model  involves  a determination  of  the  rules,  or 
formulae,  for  representing  element  relationships  within  the 
model.  The  elements  subordinate  to  the  same  immediate 
mediating  factor  combine  to  determine  its  worth  (a  term  used 


3 

The  subject  is  discussed  theoretically  in  Michael  F.  O'Connor 
and  Ward  Edwards,  "The  Use  of  Scenarios  in  the  Evaluation  of 
Complex  Alternatives"  (forthcoming) . 

4 

In  this  case,  the  Navy  reduced  an  original  selection  of  eight 
scenarios  to  four  and  tested  the  results  obtained  from  the 
MAU  model  by  comparing  them  with  the  results  of  a much- 
reduced  number  of  simulation  runs. 
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to  encompass  either  utility  or  effectiveness) . The  worth  of 
the  decoy  in  the  open  sea  scenario  (see  Figure  5-2)  is 
determined  by  the  aggregated  worths  of  probability  of  avail- 
ability, sensor  effectiveness,  effector  effectiveness,  and 
system  reaction  time.  The  rule,  or  formula,  for  repre- 
senting the  relationship  among  them  must  validly  capture  the 
manner  in  which  they  affect  the  performance  of  the  decoy  in 
the  open  sea  scenario.  In  short,  the  combination  rules 
define  the  principles  of  aggregating  the  worths  of  elements 
throughout  the  MAU  model;  they  capture  the  value  independence 
of  the  worths  of  the  elements  or  the  nature  of  their  value 
dependency. 


Often  the  manner  in  which  the  elements  combine 
is  extremely  complex,  the  result  of  a dynamic  sequential 
process.  Nonetheless,  the  essence  of  the  process  can  some- 
times be  captured  by  fairly  simple  combination  rules.  Such 
is  the  case  if  the  worth  of  the  element  to  which  sub-elements 
contribute  increases  with  the  increasing  worth  attached  to 
different  levels  of  performance  of  the  sub-elements. 

Generally,  the  manner  in  which  sub-elements, 
say,  A,  B,  C,  and  D,  combine  to  determine  the  worth  of  an 
element,  say,  X,  can  be  captured  by  one  of  three  types  of 
combination  rules. 

1)  The  additive  rule.  If  changes  in  the  worth' 
of  element  X as  a function  of  changes  in  the  level  of 
performance  of  sub-element  B do  not  depend  on  the  level  of 
sub-element  C,  and  vice  versa,  and  if  the  same  is  true  for 
the  relationship  between  sub-elements  C and  D and  also  B and 
D,  then  we  say  that  B,  C,  and  D contribute  to  element  X 
independently  of  each  other,  and  we  use  an  additive  combi- 
nation rule  to  capture  this  value-independent  relationship. 
That  is,  the  worth  of  the  system  with  respect  to  element  X 
is  the  sum  of  the  worths  of  sub-elements  B,  C,  and  D.  For 
example,  improved  performance  of  vehicle  identification  may 
enhance  the  effectiveness  of  the  sensor  without  regard  to 
the  performance  of  emitter  identification.  The  rule  for 
expressing  mathematically  the  manner  in  which  the  effec- 
tiveness of  each  of  these  two  types  of  performance  con- 
tributes to  the  effectiveness  of  the  sensor  is  additive. 
Additive  rules  are  also  known  as  compensatory  rules,  for 
decreases  in  the  worth  of  one  sub-element  can  be  compensated 
for  by  increases  in  the  worth  of  other  sub-elements  related 
to  it  in  an  additive  manner. 

2)  The  multiplicative  rule.  Another  type  of 
relation  between  an  element  say,  X,  and  associated  sub- 
elements, say,  A,  B,  C,  and  D,  occurs  when  a certain  type  of 
dependency  exists.  Suppose  element  X is  the  sensor  effec- 
tiveness of  the  decoy  in  the  open  sea  scenario  and  sub- 
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elements  A,  B,  C and  D are,  respectively,  detection  effec- 
tiveness, localization  effectiveness,  vehicle  identification 
effectiveness,  and  emitter  identification  effectiveness  (as 
shown  in  Figure  5-3) . Sub-elements  B,  C,  and  D contribute 
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Figure  5-3 

THE  EFFECTIVENESS  OF  THE  SENSOR 
OF  THE  DECOY  IN  THE  OPEN  SEA  SCENARIO 


independently  to  the  effectiveness  of  the  sensor,  and  the 
relation  among  the  three  is  captured  by  an  additive  rule,  as 
we  have  seen.  If  the  sensor  cannot  detect  the  threat, 
however,  the  sensor  does  not  perform.  That  is,  if  detection 
effectiveness  goes  to  zero,  sensor  effectiveness  goes  to 
zero.  As  a consequence,  the  performance  and  related  effec- 
tiveness of  the  other  sub-elements  of  the  sensor  are  irrel- 
evant. A multiplicative  rule  captures  this  kind  of  value- 
dependent  relationship.  Effectiveness  of  detection  ability 
is  multiplied  by  the  sum  of  the  effectiveness  associated 
with  localization,  vehicle  ID,  and  emitter  ID  to  yield 
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sensor  effectiveness  (appropriate  re-scaling  constants  are 
explained  later) . Note  that  if  the  effectiveness  associated 
with  detection  ability  drops  to  50  percent,  the  maximum 
effectiveness  that  the  sensor  can  have  is  50  percent  of  what 
it  would  have  been  if  detection  ability  were  at  the  level  of 
100-percent  effectiveness,  and  this  50-percent  effectiveness 
occurs  only  if  the  remaining  sensor  sub-elements  are  at 
levels  of  maximum  sub-element  effectiveness.  In  this  manner, 
the  multiplicative  combination  rule  can  be  used  to  describe 
non-compensatory  processes  in  which  degradation  by  one  element 
cannot  be  compensated  for  by  an  increased  performance  of 
another . 


3)  Other  combination  rules.  This  discussion  of 
additive  and  multiplicative  rules  is  not  meant  to  suggest 
that  other,  more  complicated  relationships  do  not  exist. 

For  example,  another  type  of  combination  rule  captures  what 
is  known  as  the  "either-or"  relationship,  in  which  an  element 
achieves  its  maximum  effectiveness  when  any  one  sub-element 
attains  maximum  effectiveness  but  achieves  its  minimal 
effectiveness  only  when  all  sub-elements  attain  minimal 
effectiveness. 5 Such  sophisticated  rules  played  a negligible 
role  in  this  MAU  model.  The  question  of  when  dependencies 
are  important  enough  to  merit  in-depth  analysis  is  one  that 
is  presently  receiving  a great  deal  of  attention. 


5The  reason  for  the  multiplicative  relationship  here  is  best 
explained  by  considering  the  different  kinds  of  dependencies 
that  can  occur.  Consider  system  X,  the  effectiveness  of 
which  is  a function  of  two  performance  characteristics, 

A and  B.  Denote  the  effectiveness  of  X as  Xg,  the  effec- 
tiveness of  A and  B as  AE  and  BE,  respectively.  Maximum 
effectiveness  of  X,  A,  and  B will  be  assigned  the  arbitrary 
value  of  100  and  minimum  effectiveness  of  X,  A,  and  B the 
arbitrary  value  of  0.0. 

When  A and  B can  contribute  independently  to  the 
effectiveness  of  X,  that  is,  are  value  independent,  each  is 
given  a weight  proportional  to  its  relative  contribution. 
Suppose,  for  example,  each  can  contribute  equally  to  effec- 
tiveness. Then  each  will  be  assigned  a weight  of  .5.  If 
both  A and  B are  at  their  maximum  effectiveness,  X will 
have  an  effectiveness  of  100.  If  either  A or  B is  at  its 
minimal  effectiveness  with  the  other  element  at  its  maximum 
effectiveness,  X will  have  an  effectiveness  that  is  50 
percent  of  its  maximum.  Only  if  both  A and  B are  at  miminum 
effectiveness  is  the  effectiveness  of  X at  0.0.  This 
relationship  is  captured  by  the  additive  combination  of 
A + B to  determine  XE;  that  is,  XE  = . 5AE  + . 5BE . 
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Once  the  combination  rules  are  established — 
remember:  they  continue  to  be  amended  as  necessary  as  the 

next  two  steps  are  taken — it  is  necessary  to  establish 
element  boundaries  on  ranges  of  levels  of  performance  of 
controllable  parameters.  The  purpose  of  such  boundaries  is 
twofold:  first,  they  limit  the  range  of  performance  to  be 

measured  to  plausible  dimensions;  and  second,  they  permit 
the  importance  of  that  performance  to  be  gauged  realistically 
Too  often  the  importance  attached  to  a specific  kind  of 
performance  is  inflated  because  it  is  considered  universally, 
without  regard  to  the  range  of  technically  and  economically 
feasible  levels  of  performance  likely  to  be  suggested  by 
design  proposals.  This  is  one  point  at  which  the  impact 
of  a design-to-cost  procurement  policy  is  felt.  Thus, 
whereas  an  expert  may  wish  the  sensor  of  an  EW  suite 


(5  cont.)  Now  suppose  that  if  either  A or  B is  at  its 
minimum  effectiveness,  the  effectiveness  of  X is  (0.0). 
Further,  degradation  of  either  element  degrades  overall 
effectiveness  proportionally.  If  AE  is  50,  the  maximum 
effectiveness  X can  have  is  .5,  and  then  only  when  BE  is 
at  its  maximum  effectiveness.  If  AE  is  0.0,  X has  an 
effectiveness  of  0 no  matter  what  BE  is.  This  dependency 
is  captured  by  making  the  effectiveness  of  X a function  of 
the  product  of  factors  A and  B;  that  is,  XE  = AE  x B^, . 

Thus  far,  the  only  way  that  X can  have  maximum  effec- 
tiveness is  if  A and  B are  both  at  maximum  effectiveness. 

If  an  additive  relationship  exists,  the  only  way  for  XE  to 
be  at  0.0  is  for  both  AE  and  BE  to  be  0.0.  If  a multi- 
plicative relationship  exists,  XE  is  0.0  if  either  AE  or 
Be  is  0.0.  Now  suppose  that  X can  be  at  its  maximum  effec- 
tiveness of  100  if  either  A or  B is  at  maximum  effectiveness 
A multiplicative  combination  of  the  following  sort  can 
capture  such  a relationship: 

X = [100  - (100~A) (100-B)  ] 

E 100 

Note  that  if  either  AE  or  B£  has  a value  of  100,  XE  has  a 
value  of  100,  no  matter  whar  the  value  of  the  other  element. 
XE  is  0.0  only  if  both  AE  and  BE  are  0.0.  If  both  AE 
and  Be  are  50,  XE  is  75  percent  of  its  maximum  effectiveness 

This  "either-or"  multiplicative  rule  is  a specific 
instance  of  the  more  general  formulation  of  the  multiplica- 
tive utility  function  discussed  by  R.  L.  Keeney,  "Multi- 
plicative Utility  Functions,"  Operations  Research,  22 
(1974),  22-34.  The  mathematically  inclined  reader  should 
consult  this  article  for  a more  detailed  discussion. 
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to  have  a large  capacity  to  detect  discrete  targets  and 
avoid  saturation,  he  may  set  a range  of  targets  prior  to 
saturation  from  zero  to  25  and  make  other  judgments  with 
respect  to  that  range,  like  those  for  effectiveness  curves 
and  weights.  However,  if  design  proposals  drawn  up  in  light 
of  cost-per-unit  ceiling  have  capacities  from  four  to  ten 
targets  prior  to  saturation,  it  is  unlikely  that  the  expert 
will  find  much  difference  among  the  proposals  in  this  respect 
or  attach  the  proper  degree  of  importance  to  it.  Accordingly, 
experts  must  set  boundaries  based  on  their  best  estimates  of 
what  range  is  technically  and  economically  feasible. 

Completing  the  quantification  of  the  model 
involves  two  more  iterative  steps:  developing  effectiveness 

curves  and  determining  element  weights.  It  is  necessary 
first  to  translate  measures  of  performance  of  controllable 
parameters  into  levels  of  effectiveness  expressed  as  per- 
centages, then  to  assess  the  importance  of  the  bounded 
ranges  of  performance.  The  translation  is  achieved  by 
having  the  appropriate  experts  assess  the  levels  of  effec- 
tiveness (on  a scale  of  0 to  100)  for  varying  levels  of 
performance  within  previously  defined  boundaries.  The 
result  is  an  effectiveness  curve,  a graphic  representation 
of  the  relationship  between  levels  of  performance  with 
respect  to  a mediating  factor  and  effectiveness.  An  example 
of  an  effectiveness  curve  is  shown  in  Figure  5-4.  The  signifi- 


NUMBER  OF  TARGETS  BEFORE  SATURATION 


Figure  5-4 

EFFECTIVENESS  CURVE  FOR  DETECTION 
BY  THE  SENSOR  OF  THE  DECOY  IN  THE  OPEN  SEA  SCENARIO 
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cance  of  an  effectiveness  curve  is  that  performance  is 
immediately  related  to  and  cannot  be  considered  apart  from 
worth.  The  curve  itself  means  that  performance  is  seen  as  a 
continuum  over  a range  of  levels  of  performance;  a particular 
level  of  performance  is  but  one  possibility  within  the  range 
of  plausible  levels  of  performance,  not  a point  immutable. 
Similarly,  effectiveness  varies  with  the  range  of  levels  of 
performance;  many  levels  of  performance  within  the  previously 
specified  boundaries  have  some,  though  less  than  optimum, 
effectiveness  with  respect  to  the  mediating  factor.  Effec- 
tiveness, then,  is  an  intra-parametric  measure  of  worth 
throughout  the  plausible  range  of  levels  of  performance. 

The  use  of  effectiveness  curves  has  two  benefits. 
First,  an  effectiveness  curve  avoids  categorizing  a particular 
level  of  performance  as  either  acceptable  or  unacceptable, 
as  the  stipulation  of  system  requirements  demands.  Specific 
requirements  can,  in  fact,  be  represented  by  effectiveness 
curves  in  which  effectiveness  is  a minimum  when  the  require- 
ment is  not  satisfied,  at  a maximum  when  it  is.  A specific 
requirement,  in  other  words,  stipulates  a very  special  kind 
of  effectiveness  curve.  The  absence  of  such  requirements 
permits  a more  general  use  of  effectiveness  curves,  which 
are  thus  more  flexible,  sophisticated,  and  experiential. 

The  second  benefit  from  the  use  of  effectiveness  curves  is 
that  the  particular  shape  of  an  effectiveness  curve  reveals 
those  ranges  of  performance  in  which  a slight  change  in 
performance  may  result  in  a significant  change  in  effec- 
tiveness, for  better  or  worse,  or  those  ranges  in  which  a 
significant  change  in  performance  may  result  in  little  or  no 
change  in  effectiveness.  The  effect  of  changing  levels  of 
performance  in  terms  of  TSU  can  be  readily  displayed,  and 
the  shape  of  an  effectiveness  curve  may  often  be  diagnostic 
of  that  range  in  which  a trade-off  between  performance  and 
cost  is  desirable. 

The  final  step  in  this  highly  iterative  process 
in  developing  a MAU  model  is  to  determine  the  relative 
importance  to  be  assigned  to  sub-elements  of  the  same  com- 
ponent. As  part  of  the  quantification  of  the  model,  relative 
importance  is  expressed  as  a decimal  between  0.0  and  1.0. 

The  figure  indicating  importance  is  called  a weight,  and 
the  determination  of  relative  importance  is  called  weighting. 

The  purpose  of  weighting  is  to  recognize  the 
different  contributions  that  sub-element  effectiveness  makes 
to  element  effectiveness.  Otherwise,  the  assumption  is  that 
the  effectiveness  of  each  sub-element  contributes  equally  to 
the  effectiveness  of  the  element — which  assumption  is  usually 
false. 
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The  problem  in  weighting,  then,  is  a methodological 
one.  Since  weights  impinge  upon  the  figures  for  effectiveness 
and  the  formula  for  aggregating  them,  the  difficulty  is 
twofold:  how  to  assess  weights  and  how  to  incorporate  them 

into  existing  but  as  yet  incomplete  combination  rules  for 
aggregating  measures  of  effectiveness. 

Assessing  weights  depends  upon  the  relative 
influence  a change  in  the  effectiveness  of  a sub-element 
has  upon  the  effectiveness  of  an  element,  an  influence  that 
may  be  affected  by  the  relationship  among  the  sub-elements. 

That  is,  weights  assigned  to  sub-elements  are  a function  of 
their  effect  upon  the  element  as  the  effectiveness  of  each 
changes  from  minimum  to  maximum.  As  the  effectiveness  of  B, 

C,  and  D each  changes  from  0.0  to  100,  the  effectiveness  of 
X changes  in  response. 

If  the  combination  rule  for  B,  C,  and  D is 
additive,  the  changes  in  X are  produced  by  changes  in  B,  C, 
and  D from  their  minimum  to  their  maximum  effectiveness,  the 
ratio  of  these  changes  enables  the  determination  of  weights 
renormalized  to  sum  to  one.  For  example,  changes  in  the 
effectiveness  of  B,  C,  D from  0.0  to  100  result  in  changes 
of  the  effectiveness  of  X of  20,  5,  and  5,  respectively,  the 
ratio  can  be  renormalized  to  sum  to  one,  with  weights  of 
.666,  .167,  and  .167,  respectively. 

If  the  combination  rule  for  elements  is  multi- 
plicative, say,  A x (B+C+D) , weights  are  developed  differently. 
Since  B,  C,  and  D are  combined  additively  and  together  are 
combined  with  A multiplicatively , weights  cannot  be  used 
additively  for  all  four  elements  though  they  can  be  and  are 
for  B,  C,  and  D.  The  problem  is  to  establish  some  quantified 
measure  of  relative  importance  for  elements  components 
combined  multiplicatively.  The  solution  is  to  use  what  may 
be  called  rescaling  constants  that  reflect  the  degree  to 
which  the  effectiveness  of  an  element  may  be  degraded  by  its 
sub-elements.  If  the  effectiveness  of  A,  B,  C,  and  D are  at  a 
maximum,  reducing  the  effectiveness  of  A to  its  minimum 
reduces  the  effectiveness  of  X.  Although  the  more  general 
case  does  not  require  that  when  the  effectiveness  of  A is 
zero,  the  effectiveness  of  X is  zero,  this  special  case  is 
very  common. 


For  example,  the  maximum  effectiveness  of  the 
sensor  depends  upon  the  maximum  effectiveness  of  detection, 
localization,  and  vehicle  identification  and  emitter  identi- 
fication. If  the  effectiveness  of  detection  alone  drops  to 
a minimum,  or  zero,  the  effectiveness  of  the  sensor  likewise 
drops  to  a minimum,  or  zero;  a sensor  which  cannot  detect 
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targets  is  not  performing  and,  obviously,  has  no  effectiveness. 
Accordingly,  no  rescaling  constant  is  associated  with  A;  the 
maximum  effectiveness  of  X is  limited  to  the  maximum  effec- 
tiveness of  A.  On  the  other  hand,  if  the  effectiveness  of  A 
remains  at  a maximum  and  the  effectiveness  of  B,  C,  and  D 
are  reduced  to  zero,  the  effectiveness  of  X is  reduced  to 
only  .7.  Accordingly,  no  rescaling  constant  is  added  to  A; 
if  it  goes  to  zero,  X goes  to  zero;  a rescaling  constant  of 
.7 — it  may  also  be  called  a degradation  constant — is  added 
to  the  sum  (B+C+D) . If  B+C+D  are  each  zero,  the  effec- 
tiveness of  X will  be  the  product  of  the  effectiveness  of  A 
times  the  degradation  factor.  Since  the  relative  importance 
of  the  effectiveness  of  B,  C,  and  D may  differ,  their  relative 
importance  is  established  according  to  the  procedure  out- 
lined above  for  an  additive  combination  rule.  However, 
their  weights  are  normalized  to  .3,  not  1.0,  because  of  the 
degradation  constant  of  .7.  The  completed  combination  rule 
in  this  example  (see  Figure  5-4)  is: 


Ev (sensor)  = (E,  , , . ..  + 0 . 0)  ( . 03E„  , . . + 

X A (detect)  B (localize) 

,06EC (vehicle  ID)  + * 2lED (emitter  ID)  + ,7)* 


When  other  combination  rules,  reflecting  the  nature  of  the 
relationships  among  sub-elements,  become  more  complex, 
eliciting  the  weights  becomes  more  complex.  These  refine- 
ments are  not  discussed  in  this  report. 

Weights  assigned  to  conditioning  variables  are 
dependent  upon  the  nature  of  those  variables.  In  the  case 
of  the  MAU  model  for  EW  suite  proposals,  each  scenario  is 
assigned  two  sets  of  weights,  one  for  the  relative  proba- 
bilities of  occurrence  of  the  scenario,  the  other  for  the 
relative  importance  of  accomplishing  the  mission  in  the 
scenario.  The  weights  for  each  variable  are  multiplied, 
and  the  products  are  then  normalized  to  1.0.  The  con- 
ditional utilities  assessed  for  each  scenario  are  multi- 
plied by  the  corresponding  normalized  weights,  and  the 
products  are  combined  according  to  the  rule,  here,  additively. 
This  general  procedure  provides  for  the  correct  weighting 
of  conditional  utilities  and  the  aggregation  of  these 
utilities  throughout  the  MAU  hierarchy. 

5 . 4 Validating  the  Model 


Once  the  MAU  model  is  developed,  two  types  of  validation 
analyses  are  conducted.  The  first,  internal  validation, 
consists  of  examining  the  effects  of  variations  in  weights 
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and  effectiveness  curves  for  purposes  of  identifying  critical 
aspects  of  the  model.  It  is  typical  that  such  evaluation 
models  are  effective  despite  small  errors  in  utility  functions 
or  weights.  These  sensitivity  analyses  ensured  that  the 
same  was  true  for  this  particular  model.  It  is  also  typical 
that  such  models  are  extremely  sensitive  to  changes  in 
combination  rules  or  associated  structural  changes  resulting 
from  different  interrelationships  among  model  factors.  It 
is  the  structure  of  the  model  that  either  does  or  does  not 
capture  the  true  nature  of  the  problem.  Internal  sensi- 
tivity analyses  cannot,  however,  guarantee  this  type  of 
validity. 

The  second  type  of  validation,  external  validation, 
provides  evidence  of  the  validity  of  the  model  structure, 
for  known  systems  are  used  to  calibrate  the  model.  The 
model  should  assign  to  these  known  systems,  whose  record 
of  performance  has  been  well  documented,  values  that  are 
consistent  with  the  generally  accepted  assessments  of  the 
systems  based  upon  that  record.  If  it  does  so,  it  helps  to 
ensure  that  the  model  behaves  as  it  should.  If  not,  struc- 
tural changes  are  required. 6 

The  implementation  of  such  models  on  interactive  com- 
puters also  allows  "what-if"  type  analyses  that  can  accom- 
modate proposed  design  changes.  Similarly,  the  model  can 
actually  be  used  to  search  for  those  feasible  configurations 
that  have  maximum  utility. 7 


6In  the  actual  external  validation  of  the  EW  model,  it  was 
found  that  not  enough  credit  had  been  given  the  sensor  for 
its  ability  to  detect  false  alarms.  The  model  was  altered 
to  account  for  this  fact. 

7 

The  EW  MAU  model  was  implemented  on  computers  at  the  Naval 
Research  Laboratories,  Washington,  D.C.  and  was  used  both 
for  numerous  "what-if"  type  analyses  in  the  procurement 
stage  as  well  as  for  other  design  problems  on  related  projects. 
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6.0  EVALUATION  OF  THE  MULTI-ATTRIBUTE  UTILITY  APPROACH 


6 . 1  The  Multi-Attribute  Utility  Approach  and  the  Criteria 
for  Evaluation 


Correctly  implemented,  the  MAU  approach  meets  the  four 
general  criteria  set  forth  for  evaluations.  The  approach 
used  in  this  case  is  designed  to  discriminate  among  pro- 
posals in  terms  of  TSU.  The  approach  is  reliable,  yielding 
reproduceable  results  from  the  same  data;  differences  that 
arise  among  different  sets  of  evaluators  can  be  resolved  by 
means  of  discussions  about  model  structure,  combination 
rules,  boundaries  of  ranges  of  levels  of  performance  levels 
of  controllable  parameters  of  performance,  effectiveness 
functions,  or  weights.  A major  strength  of  the  approach  is 
its  intelligibility.  The  logic  by  which  data  are  translated 
into  measures  of  utility  or  effectiveness  is  explicit  and 
subject  to  challenge  and  modification.  The  procedure  is 
equitable,  since  the  model  is  not  biased  against  any  kind 
of  design  proposals.  The  approach  also  satisfies  the  other 
specific  criteria  set  forth.  It  leads  to  models  which 
develop  unambiguous,  quantified  measures  of  utility.  The 
approach  itself  is  comparatively  inexpensive  and  is  con- 
ducive to  inexpensive  "what-if"  and  sensitivity  analyses. 

6 . 2  Ultimate  Benefit  of  the  Multi-Attribute  Utility  Approach 

Of  the  benefits  derived  from  using  the  MAU  approach, 
the  immediate,  practical  one  was  that  it  served  its  purpose 
in  helping  the  U.S.  Navy  select  two  from  among  six  con- 
tractors to  proceed  with  engineering  development  of  proto- 
types for  testing  in  a systematic  and  methodologically  sound 
manner.  More  general  benefits  from  using  the  MAU  approach 
can  be  summarized  by  stating  that  it  effectively  meets  all 
requirements  earlier  set  forth  as  either  mandatory  or  desirable 
for  the  evaluation  of  complex  systems.  In  this,  it  is  more 
satisfactory  than  the  global  approach  or  an  exclusive  re- 
liance upon  simulations  runs.  Moreover,  the  MAU  approach 
has  the  desirable  property  that  the  complexity  of  its  modeling 
can  be  tailored  to  the  need  for  complexity  in  analyzing  the 
problem.  The  emphasis  has  been  on  the  controllable  para- 
meters of  performance  as  opposed  to  all  parameters  of  per- 
formance. Certain  aspects  of  a problem  usually  are  nearly 
constant  for  all  options.  Accordingly,  these  aspects 
require  little  in  the  way  of  evaluative  effort,  and  those 
that  do  require  effort  can  have  it  allocated  optimally  from 
a cost  effectiveness  point  of  view. 

Perhaps  the  major  benefit  to  be  derived  from  the  MAU 
approach,  however,  is  that  the  approach  effectively  organizes 
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large  amounts  of  information  into  results  that  can  be  used 
effectively  by  evaluation  board  members.  It  is  not  uncommon 
that  results  of  many  large  evaluations  have  been  ignored 
because  they  could  not  be  summarized  in  the  concise  fashion 
that  is  necessary  for  high-level  decision  makers.  This 
failure  often  results  from  a lack  of  intelligibility  owing 
to  a high  degree  of  (perhaps  unnecessary)  complexity. 

A model  has  truly  done  its  job  if  it  can  be  discarded 
when  the  evaluation  effort  is  finished  and  the  evaluators 
can  summarize  the  results  of  the  model  or  explain  the  final 
decision  in  a clear,  yet  well-documented,  manner. 1 MAU 
models  succeed  in  just  this  way.  At  the  same  time,  they  are 
flexible  enough  to  allow  decision  makers  and  evaluators  to 
introduce  last  minute  questions  and  conduct  "what-if"  type 
analyses.  Because  of  these  strengths,  the  MAU  approach  is 
enjoying  increasing  popularity  as  an  evaluation  technique. 
Shepard  has  said  that  "the  optimum  choice  (out  of  a given 
set  of  alternatives)  is  the  one  that  leads  to  the  highest 
subjective  evaluation  of  its  ensuing  consequences."* 2  Such 
a definition  would  seem  to  ignore  cost,  but  the  point  is 
nevertheless  relevant.  The  choice  of  an  evaluation  approach 
for  large  complex  systems  is  an  exceedingly  important  one, 
and  the  MAU  approach  has  many  apparent  benefits,  but  the 
ultimate  relative  value  of  the  approach  will  be  determined 
as  the  results  of  its  more  widespread  use  are  observed. 

Until  such  time,  evaluation  must  proceed,  as  it  has  in  this 
report,  according  to  reasonable  criteria  and  desiderata  for 
evaluative  approaches. 


^As  a result  of  this  stage  of  the  evaluation,  the  Navy  was 
able  to  explain  its  approach,  to  debrief  the  contractors, 
and,  in  particular,  to  detail  the  strengths  and  weaknesses 
of  their  design  proposals. 

2 

R.N.  Shepard,  "On  Subjectively  Optimum  Selections  Among  Multi- 
Attribute  Alternatives,"  in  Human  Judgments  and  Optimality, 
edited  by  M.W.  Shelly  and  G.L.  Bryan  (New  York:  Wiley  and 

Sons,  1964);  reprinted  in  Decision  Making:  Selected  Readings, 

edited  by  Ward  Edwards  and  Amos  Tver sky,  Penguin  Books  (Harmonds- 
worth:  Penguin  Books,  1967),  pp.  258-59. 
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7.0  CONCLUSIONS 


7 . 1 Implications 

Although  the  development  and  use  of  the  MAU  model  was 
deemed  sufficient  in  the  case  of  the  Navy's  EW  System,  it  is 
apparent  that  additional  research  is  needed  to  develop 
prescriptive  principles  for  specifying  the  procedures  for 
selecting  scenarios  to  be  included  in  the  model  and  for 
selecting  appropriate  combination  rules,  that  is,  the 
formulae  capturing  the  relationships  among  the  elements  of 
the  model. 

Since  the  choice  of  scenarios  has  not  yet  been  sys- 
tematized, further  research  is  required  to  ensure  that  the 
selected  scenarios  are  both  representative  of  the  population 
of  possible  scenarios  and  that  they  discriminate  among  the 
alternatives  being  evaluated.  These  prerequisites  for 
scenarios  suggest  that  a stratified  sampling  procedure  would 
be  useful  in  the  process  of  defining  or  selecting  the  scenarios. 
Stratification  would  first  specify  scenarios  that  discrimi- 
nate maximally  among  the  alternatives  being  evaluated  and 
then  assign  relative  probabilities  and  weights  to  the  scenarios 
so  that  they  may  be  weighted  in  accordance  with  their  proba- 
bility of  occurrence  and  their  relative  importance. 

Second,  most  designers  of  MAU  models  have  tended  to  use 
additive  rather  than  multiplicative  combination  rules.  This 
tendency  likely  represents  a propensity  to  start  with  the 
simplest  possible  combination  rules  and  to  move  to  more 
complex  rules  only  as  a last  resort.  However,  multipli- 
cative rules  are  more  appropriate  in  cases  in  which  the 
effectiveness  of  an  element  goes  to  zero  if  the  effective- 
ness of  any  sub-element  goes  to  zero,  as  in  this  MAU  model 
of  TSU  of  EW  suite  design  proposals.  Other  more  complex 
combination  rules  may  be  more  appropriate  for  capturing  more 
complex  relationships.  Research  is  required  to  identify 
when  more  complex  rules  to  capture  combinations  of  elements 
are  necessary  or  when  simpler  approximations  are  sufficient. 

7.2  Afterword 


The  Navy's  use  of  this  evaluating  mechanism,  or  MAU 
model,  reflected  the  first  implementation  of  DoD's  design- 
to-cost  procurement  policies  from  the  inception  of  a program 
to  develop  a sophisticated  and  complex  military  system.  The 
MAU  model  served  to  select  two  proposals  from  among  six  for 
engineering  development  and  testing  of  prototypes.  Perhaps 
even  more  important,  the  use  of  the  MAU  model  affected  the 
kind  of  EW  suite  designs  which  the  Navy  selected.  Certain 
initial  considerations,  like  five  suites,  with  one  especially 
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adapted  for  airborne  use,  were  eliminated  as  of  negligible 
military  value.  The  use  of  the  MAU  model  has  not  stopped  at 
this  stage  in  the  program.  Modifications  in  the  original 
model  have  led  to  a revised  version  that  will  be  used  to 
evaluate  the  two  prototypes  when  they  are  tested  and  to 
select  the  better  one. 

Nor  is  it  likely  that  the  use  of  MAU  models  will  be 
confined  to  the  evaluation  and  selection  of  design  proposals 
and  prototypes.  It  should  be  clear  that  a MAU  model  can  be 
used  not  only  for  these  purposes,  but  also  for  aiding  in  the 
design  of  a system  and  in  diagnosing  areas  in  which  tech- 
nological development  is  worthwhile.  It  is  therefore, 
likely  that  MAU  models  will  be  employed  by  contractors  to 
assist  in  the  design  of  other  sophisticated  and  complex 
systems . 
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